Repository navigation
fix(server): Enforce request timeouts on servers that do not listen - #49
Merged
Merged
Conversation
…d do not report aborted requests Node checks `headersTimeout` and `requestTimeout` only after the `listening` event. The tunnel server never listens, and with `proxyProtocol` the HTTP server does not either, so their requests had no time limit. Send the event to them, and give headers 10 s. Node checks each second. A client that closes its connection before its body came gave an unexpected ECONNRESET, so it went to Sentry. It now gets status 400 and counts as `roach.request` with `roach.result:aborted`. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Keep both sides of #48 and of this branch: - server.ts: `readBody` clears the 30 s read deadline on end and on an error, and an ECONNRESET of a request still becomes the 400 `aborted` error. `answerError` counts `aborted`, `busy` (503), and `timeout` (408) as `roach.request`. The comment of the read deadline now says why it is shorter than Node's 300 s. - service.test.ts: keep both sets of helpers. `startBody` uses `openTunnel`. The aborted test also checks that a closed request gives its bytes of the budget back. - README.md: one metric row with all results. Remove the gap about Node time limits, because this branch fixes it. - TELEMETRY.md: Busy, Timeout, and Aborted, and leave all three out of the hit rate.
With 32 MiB, 4 IPv4 addresses or 4 /64s could fill the 128 MiB budget of runs without a token. With 16 MiB it takes 8. 16 MiB is still one body of the largest size, or about 25 of the largest model requests. The README said that a client with a /48 holds the budget "for 30 s at a time". A body gets HTTP 408 after 30 s, but the client can start the next one at once, so the hold does not stop.
dcramer
marked this pull request as ready for review
November 5, 2026 16:04
dcramer
added a commit
that referenced
this pull request
Nov 5, 2026
CI already runs on push to `main` (`ci.yml` has `push: branches: [main, "release/**"]`). But GitHub started no push workflows for `09f6582` (the #47 merge): no CI, no Image, no CodeQL. The Deploy run of `9b2ca6a` (#49) then saw a newer head and stopped ("main is at 09f6582 now, not 9b2ca6a. Nothing deploys."). So the head of `main` has no CI, no image and no deploy, and `:main` still points at `6c64d29` (#48). Nothing could start those runs again: neither workflow had `workflow_dispatch`. **Changes** - `ci.yml` and `image.yml`: add `workflow_dispatch`. - `image.yml`: push `:<sha>` on every event except `pull_request`, so a manual run on `main` makes the image. - `deploy.yml`: - `promote` accepts an Image run that isn't a `pull_request` (push or manual). The `branches: [main]` filter still limits Deploy to `main`. - "Wait for CI" moves to `.github/scripts/wait-for-ci.sh` and looks for the CI run of the commit with `--branch main`, not `--event push`. A manual CI run counts. A `pull_request` run of the same commit doesn't, because its branch isn't `main`. - `README.md`: "Deploy" says what to run when a push gets no workflows. **After merge:** this merge is a new push, so it should get CI, Image and Deploy normally. If it doesn't, run `gh workflow run ci.yml --ref main` and `gh workflow run image.yml --ref main`. **Checked** - shellcheck (`--shell=sh --enable=check-extra-masked-returns`) and actionlint pass. `oxfmt --check` passes. - The script ran under busybox `ash` with stub `gh`, `timeout` and `sleep`: success after two empty lists, a failed run, a run that never ends (60 polls, then exit 1), a `gh` error, and no arguments (exit 2). - The live query `gh run list --workflow ci.yml --branch main --commit <sha>` finds the push run for `9b2ca6a`, nothing for `09f6582`, and nothing for `3f11cb0` (#49's PR head, which has only a `pull_request` run). **Not changed:** the other inline shell in `deploy.yml` (head check, VM reset, service wait). It can move to scripts in the service audit. I can't confirm why GitHub missed the push. The commit message has no skip marker, and every push workflow is missing, so it looks like a GitHub delivery gap, not this repo's config. <!-- junior-request-attribution:start --> via @dcramer. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0ACFA5JBDX%3A1793830221.324719 --> -- [View Session](https://junior-prod.sentry.dev/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719/?project=4510944073809921) <!-- junior-session-footer:end -->
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Node checks
headersTimeoutandrequestTimeoutonly on a server that emitted'listening'(setupConnectionsTrackinginlib/_http_server.js). The tunnel server never callslisten(). WithproxyProtocolon, the main server does not call it either, anddeploy/gcp/main.tfalready setsproxyProtocol = true. So a client that sends headers slowly can keep a connection open with no time limit. This is true for runs with and without the write token.Changes
listenProxy()callstunnelServer.emit("listening"), andserver.emit("listening")whenproxyProtocolis on. A comment says that this depends on Node internals.headersTimeout: 10_000(Node's default is 60 s) andconnectionsCheckingInterval: 1000. 10 s is the same as the existing PROXY-header and tunnel first-byte limits.requestTimeoutstays at Node's default of 300 s.ECONNRESETon the request, not on the upstream response), the error gets status 400.answerErrordoes not send it to Sentry or log "Request failed". It counts asroach.requestwithroach.result:aborted. Before this change, any client that could start a run could create Sentry error events cheaply.aborteddefinition and hit-rate exclusion.Tests
proxyProtocol: trueservice it sends unfinished headers on the main server (after a PROXY header) and inside a tunnel, and expects 408 on both. On main, both connections are still open at 15 s. It also fails if only the twoemitlines are removed.[roach] Request failed: Error ECONNRESET. With a local Sentry sink, this branch sends 0 error events and 1abortedmetric with the tenant.pnpm checkandpnpm test:coveragepass on Node 24.22.2 (71 tests). No key code changed.Checked by hand with the new 10 s limit: a body sent over 16 s after fast headers still gets its normal answer, because the headers limit stops once headers are complete.
Merge notes
server.ts(readBody,answerError),service.test.ts, README and TELEMETRY.md. The changes are compatible. Whichever PR merges second gets a rebase.via @dcramer.
--
View Session [Sentry]